Image processing method and image processing apparatus

ABSTRACT

A CG image having a transparency parameter is superimposed on a shot image, which is an image picked up by an image-pickup device, to obtain a combined image. The combined image is displayed in a combined-image-display region. In the combined image, a mask region of the CG image is set based on parameter information used to extract a region of a hand. The transparency parameter of the CG image is set based on a ratio of the size of the region of the CG image excluding the mask region to the size of the shot image. By checking the combined image, which is displayed in the combined-image-display region, the user can set the parameter information by a simple operation.

CROSS REFERENCE OF RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.12/117,560 filed May 8, 2008 which claims the benefit of JapaneseApplication No. 2007-130435 filed May 16, 2007, both of which are herebyincorporated by reference herein in its entirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing method and an imageprocessing apparatus for combining a virtual image with a shot image toobtain a combined image and displaying the combined image.

2. Description of the Related Art

In the video production field, the chroma key technology is used inwhich a specified region is extracted from a live image picked up by avideo camera, and in which the specified region is combined with acomputer-graphic (CG) image.

Additionally, in a field of mixed reality (MR) in which a real space anda virtual space are naturally combined together such that the user doesnot feel that the real space and the virtual space are different in MR,the chroma key technology is also used. In the MR field, in order toextract only an object region from a shot image and display the objectregion, a technology has been suggested in which information isextracted from the shot image to generate, for example, a look-up table(LUT) that is a type of parameter information used to extract the objectregion.

An MR apparatus combines an image obtained in the virtual space, whichis rendered using computer graphics, with an image obtained in a realspace, which is picked up by an image-pickup apparatus, such as acamera, to obtain a combined image. The combined image is displayed on adisplay device, such as a head-mounted display (HMD), thereby presentingMR to a user.

When a CG image of a virtual object is superimposed and displayed on ashot image obtained in the real space, a combined image is generatedwithout superimposing the CG image in a region of, for example, a handof the user included in a region of the shot image. This is performedusing an object-region-extracting process. With this process, when thehand of a user wearing the HMD is extracted as an object, the hand isnot hidden by the virtual object in the combined image. Accordingly, MRthat can provide a feeling more similar to a feeling obtained in thereal space for the user can be presented.

Next, the object-region-extracting process in the related art used topresent MR to the user will be briefly described.

In order to extract an object region, color information concerning thehand is set in advance as parameter information, such as the LUT. Thecolor information concerning the object upon which a CG image is not tobe superimposed or displayed is registered, for example, in thefollowing methods. One method is capturing an image showing the objectpicked up by a camera, representing values of pixels as a distributionon a CbCr plane, and specifying a coordinate range of each axis of acolor space. Another method is sampling points for each axis in thecolor space, and registering values indicating whether or not thesampled points represent the object, which is called the LUT.

Next, a process in which the user wearing the HMD experiences an MRimage obtained using the object-region-extraction process will bebriefly described.

The image-pickup apparatus, which is the HMD worn by the user, picks upan image showing the region in front of the user wearing the HMD togenerate a shot image. The shot image includes a background image thatthe CG image is to be combined with, and an image of the hand of theuser.

After the shot image is captured, the CG image to be superimposed anddisplayed is generated. The CG image is a CG image excluding the regionof the hand of the user, i.e., the object region. By superimposing theCG image, which does not include the region of the hand, upon the shotimage, a combined image, upon which the CG image excluding the objectregion is superimposed and displayed, can be generated.

In the related art, a combined image is generated with a performance ofthe object-region-extracting process using a combination of the LUT, aprocess in which a range of luminance value is specified in a shotimage, a noise reduction process, in which pixels having a regionsmaller than a predetermined size are recognized as noise and notrendered in a CG image, and so forth. By generating the combined imagein such a manner, MR more similar to reality can be provided for a user.

As a method for setting the parameter information used to generate thecombined image using the object-region-extracting process, a method inwhich a luminance range is specified, or a method in which a setting ofthe number of pixels is specified in the noise reduction process can beused. However, use of another method is necessary. The method isspecifying a range of a shot image by dragging a mouse, and registeringcolors of pixels included in the specified region as color informationspecifying a region upon which a CG image is not to be superimposed ordisplayed in the LUT. As an example of an operation method forperforming such a registration in the LUT, a method can be used in whichcolors are specified directly in a CbCr color space.

However, the operation method was difficult for a general user, wholacks expertise or is not familiar with how to deal with colors, tounderstand the contents of the operation and a display of the colorspace for the operation. Accordingly, a general-user-friendly operationmethod has been suggested. The general-user-friendly operation method isspecifying the object upon which a CG image is not to be superposed ordisplayed, such as the hand described in the object-region-extractingprocess, using colors of the shot image while the shot image is beingdisplayed, and providing a specification of the colors for anMR-presenting apparatus.

Although an operability in the registration of colors in the LUT usingthe operation of selecting an image region in the shot image has beenimproved to some extent, the operation has a disadvantage describedbelow.

An appropriate value obtained by adjusting a parameter used to determinea region of the CG image that is not to be superimposed on the shotimage or the object region is determined by using subjective sensationof the user while the user is checking a positive effect of a setting onthe combined image. Accordingly, the user needs to repeat theregistration and deletion of the color information of the LUT so thatthe parameter can be approached to the appropriate value.

With the technology disclosed in Japanese Patent Laid-Open No.2005-228140, operations for extracting the color information from animage can be easily performed. However, because the parameter needs tobe slightly adjusted as the operations are in progress, an appropriateimage display is necessary in accordance with the progress of theoperations. There is no image-displaying method for displaying the shotimage, the combined image, and an image to be processed such that theseimages can be related to one another. For a slight adjustment of theparameter, the user needs to have a task of switching the operations orthe displays of the images one by one.

The registration or deletion of the color information of the LUT is anoperation in which a slight adjustment is necessary. However, when acombined image is generated using both settings of other parameterinformation and the color information of the LUT, it is necessary thatthe generation of the combined image be performed so that images with anoverall positive effect caused by both the settings of other parameterinformation and the color information of the LUT can be simply appliedto the combined image. However, there is no technology in which an imagedisplay can be changed on the basis of a mask region of a CG image whilethe combined image is being generated using the positive effect of thecurrent settings.

When the color information is extracted as the parameter informationused in the object-region-extracting process from the shot image, a userinterface (UI) for user operations is used because, with the UI, theuser can check a positive effect on the combined image on an imagedisplay, whereby the parameter can be approached to an appropriatevalue. However, because of the above-described technical problems, theUI has disadvantages described below.

There is no image-displaying method for displaying the shot image, thecombined image, and an image to be processed such that these images canbe related to one another in accordance with the progress of theoperations for extracting the color information from an image. For anadjustment of the parameter, the user needs to have a task of continuingthe operations while alternately switching and watching displays of theshot image, the combined image, and an image to be processed.

Furthermore, the user needs to have a task of performing an adjustmentof the parameter while checking the overall positive effect caused bythe settings of parameter information on a display of the combinedimage. Additionally, the user needs to have a task of manually changinga transparency parameter of a CG image in accordance with the progressof the adjustment of the parameter such that the transparency parametercan be set to an appropriate value.

SUMMARY OF THE INVENTION

The present invention provides an image processing method in which auser can use parameter information, which is used to extract an objectfrom an image, by a simple operation.

The present invention also provides an image processing method in whichan appropriate image display can be performed when the parameterinformation is to be registered.

Furthermore, the present invention also provides an image processingmethod in which a combined image, which is obtained by combining a CGimage with a shot image, can be displayed in a state in which thetransparency of the CG image is dynamically changed by the transparencyparameter. Thus, the user can easily perform operations for adjustingthe parameter information, and regions can be easily distinguished fromone another in the image.

According to an aspect of the present invention, an image processingmethod for combining a virtual image with a shot image to obtain acombined image and displaying the combined image is provided. The imageprocessing method includes the following: capturing a shot image pickedup by an image-pickup unit; extracting a predetermined region from theshot image with an extracting unit; generating a virtual image based ona transparency parameter based on a ratio of the size of thepredetermined region to the size of the shot image; combining thevirtual image with a region of the shot image excluding thepredetermined region to obtain a combined image; and displaying thecombined image.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention and,together with the description, serve to explain the principles of theinvention.

FIG. 1 is a block diagram of a configuration of an image processingsystem according to an embodiment.

FIG. 2 is an illustration of an example of a user interface according tothe embodiment.

FIG. 3 is a flowchart of processing of operations in the imageprocessing system according to the embodiment.

FIG. 4 is an illustration of an example of a user interface according toan embodiment.

FIG. 5 is an illustration of an example of a user interface according toan embodiment.

DESCRIPTION OF THE EMBODIMENTS

Embodiments of the present invention will be described below in detailwith reference to the accompanying drawings.

FIG. 1 is a block diagram of a configuration of an image processingsystem according to an embodiment.

Referring to FIG. 1, an image processing apparatus 1 includes aninformation processing device 10, an HMD 30, and aposition-and-orientation-measurement section 50. The informationprocessing device 10 includes a processing section 100, a display unit200, and an operation-input unit 400. Theposition-and-orientation-measurement section 50 includes asensor-control unit 500 and a sensor 600.

The processing section 100 is implemented by a personal computer or thelike. More specifically, the processing section 100 includes a centralprocessing unit (CPU) 101, a memory unit 102 having a random-accessmemory (RAM), a read-only memory (ROM), and so forth, animage-generating unit 103, a system bus 104, a disk drive 105, an inputunit 106, and an image-capturing unit 107.

The CPU 101 exercises overall control of the image processing apparatus1 on the basis of, for example, a control program for achievingfunctions described below, which is stored in the memory unit 102. TheCPU 101 is connected through the system bus 104 to each of the memoryunit 102, the image-generating unit 103, the disk drive 105, the inputunit 106, and the image-capturing unit 107 so that the CPU 101 and eachof these units can communicate with each other.

The memory unit 102 serves as a main memory. The memory unit 102temporarily stores the program code of the control program, controlinformation concerning the control program, CG image data,measurement-position data, and so forth via the system bus 104.

The image-generating unit 103 is implemented by a device such as a CGcard. Generally, the image-generating unit 103 includes a CG memory notshown in FIG. 1. The CPU 101 executes a program to generate imageinformation, and the image information is written into the CG memory ofthe image-generating unit 103 via the system bus 104. Theimage-generating unit 103 converts the image information written in theCG memory to an image signal suitable for a display unit 301, and sendsthe image signal to the display unit 301. The CG memory is notnecessarily included in the image-generating unit 103, and functions ofthe CG memory may be implemented in the memory unit 102.

The units constituting the processing section 100 are connected to thesystem bus 104, and communicate with one another via the system bus 104.

The disk drive 105 is implemented by an auxiliary memory, such as a harddisk. The disk drive 105 stores the program code of the control program,the control information concerning the control program, CG object dataof a virtual space, position data of the HMD 30, parameter informationused in an object-region-extracting process, and data on a setting. Thedisk drive 105 also stores data on a shot image obtained from theimage-capturing unit 107, a parameter for the control of a system, andso forth.

The input unit 106 is implemented by various types of interface devices.For example, when a signal is input as data from a device, which isexternally connected to the processing section 100, in the input unit106, the input unit 106 realizes a function of writing the data throughthe system bus 104 to the memory unit 102. Additionally, the input unit106 accepts an instruction provided through the operation-input unit 400by a user or an input from an external device.

In the image-capturing unit 107, an image signal of a video image or thelike is input, as with a capture card. The image-capturing unit 107writes image data into the memory unit 102 or the CG memory via thesystem bus 104. The image-capturing unit 107 also writes image data intothe disk drive 105 so that the disk drive 105 can store the image dataas image data used in this embodiment.

The display unit 200 includes a display device such as a cathode-raytube display or a liquid-crystal display (LCD). For a user of the imageprocessing system according to the embodiment, the display unit 200displays operating states of the whole system, a UI for user operations,HMD-position information, and so forth.

The HMD 30 is a head-mounted display that the user of the imageprocessing apparatus wears to experience MR. In the HMD 30, the displayunit 301, image-pickup devices 303, and a measurement object 302 whoseposition and orientation are measured are mounted. Although aconfiguration of the HMD 30 is described in this embodiment in whichonly one HMD is used, the configuration can be applied to an embodimentin which a plurality of HMDs are used as in the case of this embodiment.

The display unit 301 includes, for example, two sets of an LCD and anoptical system. The two sets of an LCD and an optical system aredisposed so as to be individually positioned in front of the right andleft eyes of the user. The display unit 301 is used to display imagessent from the image-generating unit 103 to present MR to the user.

The image-pickup devices 303 are used to pick up images of a real spacethat can be observed from the position of a viewing point of the user ofthe HMD 30. The pickup images are sent as image signals to theimage-capturing unit 107.

The measurement object 302 is necessary for the sensor 600 to recognizea position of the HMD 30, and is provided on the HMD 30.

The operation-input unit 400 includes, for example, devices such as akeyboard and a mouse. With the operation-input unit 400, the user canperform operations and issue instructions for the image processingapparatus. The operation-input unit 400 is used mainly to provideoperations and instructions such as operations and instructions foractivating the system, for ending the system, for controlling thesystem, and for changing settings, excluding operations for controllinga virtual object during the experience of MR.

The sensor 600 measures a position and orientation of the user of theHMD 30. The sensor-control unit 500 controls the sensor 600 to acquiremeasurement data using the measurement object 302 and the sensor 600.The position-and-orientation-measurement section 50 is configured toperform the function thereof using the measurement object 302, thesensor-control unit 500, and the sensor 600.

The sensor-control unit 500 calculates information concerning a positionand orientation of the measurement object 302 relative to apredetermined reference position on the basis of the measurement data.The sensor-control unit 500 sends the information as a measured value tothe input unit 106.

With the above-described configuration, the image processing apparatuscan present MR achieved using the object-region-extracting process tothe user, and can also provide an image display according to theembodiment.

Next, as an example of an operation of setting parameter informationused to extract an object region for a presentation of MR, an examplerelated to an image display of the display unit 200 and an operationusing the operation-input unit 400 will be described. Whether or not thesensor 600 is provided, this example can be employed.

The example can also be employed when the sensor 600 measures theimage-pickup devices 303 instead of measuring the measurement object 302as a measurement object.

More specifically, a process will be described in which, in order to setthe parameter information used to extract an object region, a combinedimage is displayed while a transparency parameter of a CG image is beingchanged in accordance with an operation procedure.

FIG. 3 is a flowchart of processing of operations according to theembodiment.

FIG. 3 shows a process in which a shot image is picked up, and in whicha combined image is displayed on the display unit 200 when the user ofthe system extracts the object region, in a system configuration shownin FIG. 1.

As an example in which the combined image is displayed, a UI is shown inFIG. 2. The design of the UI is not limited particularly thereto. Inthis embodiment, as the operation using the operation-input unit 400, anoperation using a mouse of a personal computer is described. However,instead of using the operation-input unit 400, other operation devicessuch as a keyboard and a gamepad can be used when the same operation canbe performed with the operation devices.

Referring to FIG. 3, the image processing apparatus is in a state inwhich an initialization process has been finished for the whole imageprocessing apparatus.

When the user issues an instruction for capturing shot images using theoperation-input unit 400, in step S301, the image-pickup devices 303pick up the shot images. Then, the image-capturing unit 107 captures theshot images. The shot images that have been captured are stored in thememory unit 102 or the disk drive 105.

Which device captures the shot images in step S301 is determined on thebasis of a purpose of the system. The image-pickup devices 303 in theHMD 30 need not be used necessarily. An image-pickup device, such as ageneral camera, may be used. Additionally, a plurality of shot imagesmay be captured to be optionally used if necessary.

Next, in step S302, the parameter information used to extract the objectregion is set. The parameter information can be set, for example, usingthe UI shown in FIG. 2. When a default value is maintained in the systemor when a certain value has been already set, a setting stored in thesystem, such as the default value or the certain value, can be used asit is. In such a case, this step may be omitted. Additionally, in thisembodiment, a hand of the user is displayed as an object occupying apredetermined region. However, other portions, such as a head, and anexternal portion specified by a specified color, may be used.

Next, FIG. 2 will be described. A parameter-setting user interface 2001used to set the parameter information, which is displayed on the displayunit 200, is shown in FIG. 2.

Referring to FIG. 2, a button 2002 is a button used to issue theinstruction for capturing the shot images. The user selects the button2002, for example, using a mouse. By selecting the button 2002, the shotimages are captured from the image-pickup devices 303 that are disposedindividually on the right and left sides in the HMD 30. Reduced imagescorresponding to the shot images are displayed in a region 2004 (referto step S301).

One of the shot images displayed in the region 2004 (an image on theleft side in this embodiment) is shown in a combined-image-displayregion 2003. Additionally, in the combined-image-display region 2003, aCG image having a color specified in a color region 2005 is superposedand displayed. The CG image is stored in a stencil buffer that is aregion of the memory unit 102 and that has the same size as that of theshot image. As will be described later, the CG image is displayed in astate in which the transparency of the CG image is changed by thetransparency parameter in accordance with the object-region-extractingprocess. At first, the CG image is displayed in a state with a hightransparency. The user can move a slide bar 2006 to set the transparencyparameter of the CG image.

Referring back to FIG. 3, in step S303, with reference to defaultparameter information, which has been already set in the case of thesecond or later object-region-extracting process, used to extract theobject region, a display region and a non-display region (a mask region)of the CG image are set in the stencil buffer. Various types ofparameters, such as an LUT, a range of luminance value of the image, anda parameter for the deletion of a region of the CG image whose size isequal to or smaller than a fixed size, can be used to determine theregions of the CG image when a combined image is generated.

When the regions of the CG image are set in the stencil buffer, in stepS304, a ratio of the size of the mask region of the CG image, which isgenerated in the stencil buffer, to the size of the shot image iscalculated. Alternatively, a ratio of the size of the CG image to thesize of the shot image may be used.

In step S305, on the basis of the ratio of the size of the mask regionof the CG image to the size of the shot image, the transparencyparameter of the CG image is set. In a general method, as a parameterfor the determination of the transparency of the CG image, an α value ofa color is set in a range of 0 to 1.0. In step S305, when the size ofthe mask region, which is generated using the LUT used to extract theobject region and is set in the stencil buffer, is as large as the sizeof the shot image, a parameter indicating a high transparency (a valueclose to 1.0), which does not indicate a completely transparent state,is set. When the size of the mask region of the CG image is small (avalue close to 0), a parameter indicating a low transparency is set.

As described above, the transparency parameter of the CG image can bechanged on the basis of the size of the CG image excluding the maskregion when the combined image is to be displayed. Accordingly, when thecombined image is initially displayed, the transparency parameter of theCG image is set to be high because the size of the region of the CGimage is large.

The setting of the transparency parameter of the CG image has beendescribed above. Regarding settings for the CG image, when thetransparency parameter is set, color information is generally set at thesame time. For example, for image data that needs a determination ofcolors of the CG image obtained by adjusting red, green, blue, and alpha(RGBA) values, an α value and the colors of the CG image are set at thesame time. In this embodiment, arbitrary colors can be set as the colorsof the CG image, and the colors of the CG image are not limited thereto.In this step, the colors of the CG image may be changed to colors easyto be distinguished from colors of the shot image, or colors moreappropriate for an operation purpose. Additionally, when thetransparency parameter is set on the basis of the ratio of the size ofthe CG image, which is to be superimposed on the shot image, a methodmay be used in which the colors of the CG images are changed incombination with the determination of the transparency parameter so thatthe colors of the CG image can be changed on the basis of thetransparency parameter and an operation purpose.

As a method for generating a translucent image, there are variousmethods including a method in which a translucent texture is prepared.Instead of using the above-described method, any method for generatingand displaying a translucent image can be used in the embodiment of thepresent invention as long as the transparency parameter can be set onthe basis of the ratio of the size of the CG image to the size of theshot image in the method.

Next, in step S306, the translucent CG image is generated using the CGimage stored in the stencil buffer, the transparency parameter, and thecolor information. The generated translucent CG image is a CG imageexcluding the object region (the hand of the user in this embodiment),which is extracted on the basis of the parameter information that hasbeen set.

In step S307, the combined image is generated by rendering the shotimage, which is captured in step S301, and the CG image, which isgenerated in step S306, in a frame buffer of the image-generating unit103. In step S308, the combined image is displayed on the display unit200. For example, the combined image is displayed as shown in thecombined-image-display region 2003 of FIG. 2.

In step S309, in order to change the setting of a parameter such as thesetting of the transparency parameter of the CG image and the setting ofthe color information, the process from step S303 to step S308 is to berepeated. In this case, in the object-region-extracting process, as thesetting is approached to a setting of a parameter with which the objectregion can be properly extracted from the shot image, the ratio of thesize of the mask region of the CG image to the size of the shot image isincreased.

Normally, when only the object region, for example, only the regionhaving the color information concerning the hand of the shot image, isextracted from the whole region of the shot image, the CG image is notsuperposed or displayed in the region of the hand. However, when thesetting of a parameter is not appropriate, the object region of thecombined image is hidden by the CG image.

In other words, there is a non-extracted region of the object region(with a skin color of the hand) of the shot image, and the CG image issuperimposed and displayed in the non-extracted region of the hand ofthe combined image. In order to extract color information that has notbeen extracted and register the color information in the LUT, the useradditionally registers the color information concerning thenon-extracted region in the object region of the shot image in the LUT.

In order to specify the non-extracted region of the object, the user canselect the above-described region of the combined image which isdisplayed in the combined-image-display region 2003 and in which the CGimage is superimposed and displayed, whereby an operation for theadditional registration can be easily performed.

More specifically, the non-extracted region, such as a region 4001 shownin FIG. 4, can be specified by controlling a mouse or the like on theobject image on which the CG image is superimposed and displayed. Whenthe transparency parameter is high, the operation of specifying a regionof the object that is still hidden by the CG image can be easilyperformed. For example, when most of the shot image is hidden by the CGimage, i.e., when the size of the mask region is small, the CG image isdisplayed with a high transparency as shown in FIG. 4.

When the object-region-extracting process is in progress, the extractionof the object region and the resetting of a parameter are repeated. Insuch a case, the non-extracted region of the object that is still hiddenby the CG image can be made very small. The non-extracted region of theobject is approached to an edge between the object and a region in whichthings other than the object are displayed. As a result, the user needsto perform an operation for a smaller region.

In this case, because the CG image is displayed with a low transparency,the non-extracted region is clearly displayed in the object region asshown in FIG. 5. Accordingly, in order that the color information thathas not been extracted is extracted from the image region of the hand,which is the object, and is applied to the mask region, a region havingthe color information that has not been extracted can be easilyspecified by a mouse or the like as shown in a region 5001 of FIG. 5.

Furthermore, when a region larger than the object region is accidentallymasked by the CG image, the color information registered in the LUT canbe removed from the LUT. More specifically, the color informationconcerning the background excluding the object can be specified by thesame user operation as that used in a case in which the colorinformation has been added to the LUT. Accordingly, the colorinformation that is unnecessary for the generation of the mask region ofthe object can be removed from the LUT. In this case, because the sizeof the mask region is also increased and approached to the size of theshot image, the

CG image is displayed with a low transparency. Accordingly, the CG imagecan be clearly distinguished from the shot image in the backgroundexcluding the object.

When the adjustment of a parameter with which the object region can beproperly extracted is finished (Yes in step S309) after the resetting ofthe parameter is repeated, the UI is closed to finish the image display.

Before the image display is finished, depending on a specification ofthe UI performing a method for providing the image display, the settingmay be saved in a file. Alternatively, the setting is maintained in thesystem, and for example, only the setting may be used when theMR-presenting apparatus displays an MR image.

An example in which the transparency parameter of the CG image ischanged has been described in this embodiment. However, instead of usingthe transparency parameter of the CG image, the CG image can bedisplayed using a mesh process. More specifically, in anotherembodiment, the roughness of a mesh or the number of blinks of the CGimage can be changed in the same manner as that used in the case of thetransparency parameter.

In a case of the mesh process, as in the case of a calculation methodfor calculating the transparency parameter in step S305, when the ratioof the mask region is low, the roughness of the mesh is set to be high.In contrast, when the ratio of the mask region is high, the roughness ofthe mesh is set to be low. As a result, the same positive effect as thatobtained in the case of the transparency parameter can be obtained.

In a case in which the CG image is blinked, as in the case of thesetting of the transparency parameter in step S305, when the ratio ofthe mask region is low, an interval between blinks are set to be long.In contrast, when the ratio of the mask region is high, the intervalbetween blinks is set to be short. In a combined-image-displayingprocess of step S308, the combined image is displayed at a blinkinterval that has been set. As a result, the same positive effect asthat obtained in the case of the transparency parameter can be obtained.

Other Embodiments

In another embodiment of the present invention, a computer-readablestorage medium, in which the program code of a software program capableof realizing the functions described in the foregoing embodiment isstored, is included in a system or an apparatus. A computer (a CPU or amicroprocessor unit (MPU)) of the system or the apparatus reads andexecutes the program code stored in the computer-readable storagemedium, thus realizing the functions.

The program code itself, which is read from the computer-readablestorage medium, can realize the functions described in the forgoingembodiment. Accordingly, the computer-readable storage medium, in whichthe program code is stored, can be provided in accordance with theembodiment of the present invention.

Examples of the computer-readable storage medium, which can provide theprogram code for the computer, include a floppy disk, a hard disk, anoptical disk, a magneto-optical disk, a compact disk read-only memory(CD-ROM), a compact disk recordable (CD-R), a digital-video-diskread-only memory (DVD-ROM), a digital-video-disk recordable (DVD-R), amagnetic tape, a nonvolatile memory card, and a ROM.

In the embodiment, the computer reads and executes the program code,resulting in the realization of the functions described in the forgoingembodiment. In another embodiment, on the basis of instructions of theprogram code, an operating system (OS), which operates in the computer,or the like may perform a part of or all of processes to realize thefunctions described in the forgoing embodiment.

According to the embodiments of the present invention, the combinedimage, which is obtained by combining the CG image with the shot image,can be displayed in a state in which the transparency of the CG image ischanged by the transparency parameter. Thus, the user can easily performoperations for adjusting the parameter information.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all modifications and equivalent structures and functions.

1. An image processing apparatus, for editing a parameter of extractinga region which is in association with a real space object from a realimage, with visualization of both a border line of a virtual mask imageand a border line of the real space object on the real image, the imageprocessing apparatus comprising: an image acquisition unit configured toacquire a real image obtained by capturing the real space object; aregion extraction unit configured to extract a region which isassociated with the real space object from the real image, using anextraction parameter which is set in advance; a generation unitconfigured to generate a translucent virtual mask image based on atransparency parameter; a combining unit configured to generate acomposite image of the real image and the translucent virtual maskimage, wherein the translucent virtual mask image has been clipped basedon the extracted region; a display unit configured to display thecomposite image; and a changing unit configured to change the extractionparameter based on an input operation, after displaying the compositeimage by the displaying unit.
 2. The image processing apparatusaccording to claim 1, further comprising: an operation input unitconfigured to input an operation of changing the extraction parameter,after displaying the composite image by the displaying unit; wherein thechanging unit changes the extraction parameter based on the inputoperation.
 3. The image processing apparatus according to claim 1,wherein the region extraction unit extracts a second region which isassociated with the real space object from the real image, using theextraction parameter which is changed by the changing unit; wherein thecombining unit generates a second composite image of the real image andthe virtual mask image from which the second region is excluded; andwherein the display unit displays the second composite image.
 4. Theimage processing apparatus according to claim 1, wherein the regionextraction unit extracts pixels which have a color which is indicated bythe extraction parameter from pixels which constitute the real image asthe region.
 5. The image processing apparatus according to claim 1,wherein the display unit alternately displays the real image and thecomposite image by switching between the real image and the compositeimage.
 6. The image processing apparatus according to claim 5, whereinthe display unit changes a time interval of the switching based on aratio of the size of the extracted region to the size of the real image.7. The image processing apparatus according to claim 6, wherein the timeinterval is changed to a longer time interval when the ratio is changedto a larger ratio, and the time interval is changed to a shorter timeinterval when the ratio is changed to a smaller ratio.
 8. An imageprocessing apparatus, for editing a parameter of extracting a regionwhich is in association with a real space object from a real image, withvisualization of both of a border line of a virtual mask image and aborder line of the real space object, comprising, the image processingapparatus: an image acquisition unit configured to acquire a first imageobtained by capturing the real space object by a first image-pickupunit, and a second image obtained by capturing the real space object bya second image-pickup unit; a region extraction unit configured toextract a first region which is associated with the real space objectfrom the first image using an extraction parameter which is set inadvance; a generation unit configured to generate a translucent virtualmask image based on a transparency parameter; a combining unitconfigured to generate a first composite image of the first image andthe translucent virtual mask image, wherein the translucent virtual maskimage has been clipped based on the first region; a display unitconfigured to display the first composite image; an operation input unitconfigured to input an operation of changing the extraction parameter,after displaying the first composite image by the displaying unit; and achanging unit configured to change the extraction parameter based on theinput operation; wherein the region extraction unit extracts a secondregion which is associated with the real space object from the secondimage, using the extraction parameter changed by the changing unit;wherein the combining unit generates a second composite image of thesecond image and the translucent virtual mask image wherein, thetranslucent virtual mask image having been clipped based on the secondregion; and wherein the display unit displays the second compositeimage.
 9. An image processing method, for editing a parameter ofextracting a region which is in association with a real space objectfrom a real image, with visualization of both a border line of a virtualmask image and a border line of the real space object, the imageprocessing method comprising: acquiring a real image obtained bycapturing the real space object; extracting a region which is associatedwith the real space object from the real image, using an extractionparameter which is set in advance; generating a translucent virtual maskimage based on a transparency parameter; generating a composite image ofthe real image and the translucent virtual mask image, wherein thetranslucent virtual mask image has been clipped based on the extractedregion; displaying the composite image; and changing the extractionparameter based on an input operation, after displaying the compositeimage.
 10. A non-transitory storage medium in which a program for acomputer to execute the method according to claim 9 is stored.
 11. Animage processing method, for editing a parameter of extracting a regionwhich is in association with a real space object from a real image, withvisualization of both a border line of a virtual mask image and a borderline of the real space object, the image processing method comprising:acquiring a first image obtained by capturing the real space object by afirst image-pickup unit, and a second image obtained by capturing thereal space object by a second image-pickup unit; extracting a firstregion which is associated with the real space object from the firstimage; generating a translucent virtual mask image based on atransparency parameter; generating a first composite image of the firstimage and the virtual mask image, wherein the translucent virtual maskimage has been clipped based on the first region; displaying the firstcomposite image; inputting an operation of changing the extractionparameter, after displaying the first composite image; changing theextraction parameter based on the input operation; extracting a secondregion which is associated with the real space object from the secondimage, using the extraction parameter which is changed; generating asecond composite image of the second image and the translucent virtualmask image, wherein the translucent virtual mask has been clipped basedon the second region; and displaying the second composite image.
 12. Anon-transitory storage medium in which a program for a computer toexecute the method according to claim 11 is stored.